Skip to content

New reader for G4X datasets (Singular Genomics) #281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ckmah wants to merge 17 commits into from
Closed

New reader for G4X datasets (Singular Genomics) #281

ckmah wants to merge 17 commits into from
+724 −0

Conversation

@ckmah
Copy link
Contributor

@ckmah ckmah commented Feb 19, 2025 •
edited
Loading

👋 Hello @scverse/spatialdata and community,

I would like to contribute the initial version of a spatialdata-io reader for Singular Genomics G4X datasets that I recently developed for internal use (I work at Singular), and now for the spatial community. It is still experimental and not fully battle-tested, but I tried to keep the API consistent with the other readers as much as possible. However, there are few key additions I made to streamline use with our datasets:

Notable features

  • Incremental I/O of elements (images, tables etc.) G4X datasets can get pretty large since they are multimodal. Therefore, we made sure the reader saves elements as soon as they are converted to reduce memory Reduce readers' memory consumption #229 and mitigate data loss. This is handled via the g4x(..., mode="append") parameter. The user can also choose mode="overwrite" to turn this off. The constructed SpatialData is also re-read from disk automatically to fully take advantage of lazy loading.
  • Read one or more samples at once. This corresponds to our assay design and enables converting an entire experiment with a single function call. The reader will then return a single SpatialData object or a list of them accordingly.

Additional Dependencies

  • Some of our images are encoded in the Jpeg2000 (.jp2, j2k) format and require the glympur package to read

Misc.

Are there any other pieces I should have in this PR? Devs please let me know, I'm happy to add them. Here are relevant ones I can think of:

  • Documentation/tutorial notebook? (also not sure if I used @injectdocs decorator properly)
  • Parse experimental metadata: sample names, positions, acquisition info etc.
  • spatialdata-io CLI compatibility

ckmah and others added 12 commits February 18, 2025 21:14
@ckmah ckmah mentioned this pull request Mar 24, 2025
Closed
@codecov-commenter
Copy link

codecov-commenter commented Apr 24, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 17.19298% with 236 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.41%. Comparing base (296d9a5) to head (0333b29).
⚠️ Report is 255 commits behind head on main.

Files with missing lines Patch % Lines
src/spatialdata_io/readers/g4x.py 10.26% 236 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #281      +/-   ##
==========================================
+ Coverage   39.16%   46.41%   +7.24%     
==========================================
  Files          26       27       +1     
  Lines        2663     2995     +332     
==========================================
+ Hits         1043     1390     +347     
+ Misses       1620     1605      -15
Files with missing lines Coverage Δ
src/spatialdata_io/__init__.py 100.00% <100.00%> (ø)
src/spatialdata_io/_constants/_constants.py 100.00% <100.00%> (ø)
src/spatialdata_io/readers/g4x.py 10.26% <10.26%> (ø)

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@floRaths
Copy link

floRaths commented Oct 3, 2025

Hi @LucaMarconato, I would like to pick up this PR and get it past the pre-commit checks. How do you suggest me continuing? Should I create a fresh clone of the feature on the current main branch?

@LucaMarconato
Copy link
Member

LucaMarconato commented Oct 6, 2025

Hi @floRaths, thanks for reaching out! You could open a PR against this PR, or if you want merge this PR in a new PR (from you) against main. As you prefer.

We recently added a contributed guide for spatialdata-io that will make is easier to develop PRs requiring minimal review. I kindly ask you to consult it if you want to contribute the reader: https://github.com/scverse/spatialdata-io/blob/main/docs/contributing.md.

Also, in case you want to discuss via chat or setup a meeting, we offer community meetings every 2 weeks (covering different time zones): #spatial > spatialdata meetings @ 💬.

@derrik-gratz
Copy link

derrik-gratz commented Oct 21, 2025

Just chipping in that I am currently using this branch for loading singular data with some pilot exercises and other 3rd party tools, so I appreciate the continued development :)

@LucaMarconato
Copy link
Member

LucaMarconato commented Jan 12, 2026

I recommend opening a new PR that builds on the current code (I did a quick review and it looks good; thank you, @ckmah); I'll close the existing one. The same applies to the linked PR in spatialdata-sandbox available here.

The main reason for closing is that we now have more structured process for contributions available in this guide: https://github.com/scverse/spatialdata-io/blob/main/docs/contributing.md. This new process streamlines code review on our side, facilitates continuous testing, improves long-term maintenance, and clarifies dataset licensing.

@derrik-gratz, you mentioned using this code in your workflows. Even if we close this, you can continue using @ckmah’s branch directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ckmah @codecov-commenter @floRaths @LucaMarconato @derrik-gratz